Port Decommissioning

ABSTRACT

A management application can automatically coordinate the decommissioning of ports in a switch, ensuring that ports are gracefully taken out of service without unnecessary interruption of service and triggering of automatic recovery functionality that may occur during manual decommissioning of ports. Embodiments may provide for decommissioning of F_ports only, E_ports only, or both F_ports and E_ports. Where link aggregation is provided for in the fabric, decommissioning of a port in a link aggregation group may be performed if there are other operational links in the group. Decommissioning of a non-aggregated port (or the last operational port in a link aggregation group) may be performed if there is a redundant path available.

TECHNICAL FIELD

The present invention relates to the field of computer networking, andin particular to techniques for managing ports in a switch fabric.

BACKGROUND ART

Storage area networks (SANs) are typically implemented to interconnectdata storage devices and data servers or hosts, using network switchesto provide interconnectivity across the SAN. SANs may be complex systemswith many interconnected computers, switches, and storage devices. Theswitches are typically configured into a switch fabric, and the hostsand storage devices connected to the switch fabric through ports of thenetwork switches that comprise the switch fabric. Most commonly, FibreChannel (FC) protocols are used for data communication across the switchfabric, as well as for the setup and teardown of connections to andacross the fabric, although these protocols may be implemented on top ofEthernet or Internet Protocol (IP) networks.

Typically, hosts and storage devices (generically, devices) connect toswitches through a link between the device and the switch, with a nodeport (N_port) of the device connected to one end of the link and afabric port (F_port) of a switch connected to the other end of the link.The N_port describes the capability of the port as an associated deviceto participate in the fabric topology. Similarly, the F_port describesthe capability of the port as an associated switch.

Over time, SANs have become more complex, with fabrics involvingmultiple switches that use inter-switch links (ISLs) connected to switchports (E_ports) on the switches. In some SANs, a core group of switchesmay provide backbone switching for fabric interconnectivity, with few orno devices directly connected to the core switches, while a number ofedge switches provide connection points for the devices or devices ofthe SAN. Additional layers of switches may also exist between the edgeswitches and the core switches.

The increasing complexity of enterprise data centers has complicatedswitch maintenance operations, which require coordination betweenseveral administrators in order to complete. The simple act of blockinga port of a network switch to perform a service action can requiremultiple teams of users to update their operations in order to avoidusing the target port. Often the procedures involved with this kind ofeffort are manual and prone to error.

Some environments are especially sensitive to administrative errors thatactivate the automated recovery systems. Such systems include call home,device recovery, logical path recovery, and application recoverysystems. Thus, errors in coordination of switch maintenance operationsthat trigger these automated recovery systems create delays andescalations in production environments that enterprise data centermanagers would like to avoid.

SUMMARY OF INVENTION

A management application can automatically coordinate thedecommissioning of ports in a switch, ensuring that ports are gracefullytaken out of service without unnecessary interruption of service andtriggering of automatic recovery functionality that may occur duringmanual decommissioning of ports. Embodiments may provide fordecommissioning of F_ports only, E_ports only, or both F_ports andE_ports. Where link aggregation is provided for in the fabric,decommissioning of a port in a link aggregation group may be performedif there are other operational links in the group. Decommissioning of anon-aggregated port (or the last operational port in a link aggregationgroup) may be performed if there is a redundant path available.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate an implementation of apparatusand methods consistent with the present invention and, together with thedetailed description, serve to explain advantages and principlesconsistent with the invention. In the drawings,

FIG. 1 is a block diagram illustrating a deployment of a portdecommissioning technique according to one embodiment.

FIG. 2 is a graph illustrating a state diagram for the high-levelactions that are performed to achieve port decommissioning according toone embodiment.

FIG. 3 is a flowchart illustrating a technique for decommissioning aport implemented by a wizard according to one embodiment.

FIGS. 4 and 5 are sequence diagrams that describe the decommissioningbehavior of a port decommissioning protocol according to one embodiment.

FIG. 6 is a block diagram illustrating an example switch in which aportion of the techniques described below may be implemented accordingto one embodiment.

DESCRIPTION OF EMBODIMENTS

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the invention. It will be apparent, however, to oneskilled in the art that the invention may be practiced without thesespecific details. In other instances, structure and devices are shown inblock diagram form in order to avoid obscuring the invention. Referencesto numbers without subscripts or suffixes are understood to referenceall instance of subscripts and suffixes corresponding to the referencednumber. Moreover, the language used in this disclosure has beenprincipally selected for readability and instructional purposes, and maynot have been selected to delineate or circumscribe the inventivesubject matter, resort to the claims being necessary to determine suchinventive subject matter. Reference in the specification to “oneembodiment” or to “an embodiment” means that a particular feature,structure, or characteristic described in connection with theembodiments is included in at least one embodiment of the invention, andmultiple references to “one embodiment” or “an embodiment” should not beunderstood as necessarily all referring to the same embodiment.

References to F_ports should be understood as including references toFL_ports and generic ports that can operate as F_ports. Similarly,references to E_ports should be understood as including references toEX_ports and generic ports that can operate as E_ports.

To reduce or eliminate the effort and potential for error in networkswitch maintenance operations that may create delays and escalations inproduction environments, such as may be caused by the activation ofautomated recovery systems, techniques described below provide forautomatic coordination of the removal of a port from service. Gracefulport decommissioning such as are described below allow an administratordesiring to perform a service action on a port of a network switch toremove a port from use in a way that automatically coordinates thenecessary actions to complete the decommissioning without loss of data.The port decommissioning techniques allow choosing a redundant path ofequal cost for uninterrupted data flow during the period when the portis out of service.

The techniques provide ways to identify the target port and tocommunicate the desired action to those systems that are affected by theaction. Each system involved indicates whether it agrees with thedesired action and the automated technique collects the responses beforetaking any action.

Port decommissioning generally comprises three components: (1) anapplication capable of displaying the switch fabric topology andproviding a way to select the port or ports that are to bedecommissioned, (2) processes capable of evaluating the port or ports todetermine if decommissioning is feasible and advisable, and (3) a switchfabric interface capable of blocking the selected port or ports.

In various embodiments, these components may be provided in differentways. In some embodiments, multiple entities within the network switchmanagement topology may provide some or all of the components. In otherembodiments, some or all of the components may be provided within asingle entity in the management infrastructure.

In some embodiments, port decommissioning may be performed on E_portsconnected to ISLs between network switches in the fabric. In otherembodiments, port decommissioning may be performed on F_ports connectedto links to devices. In yet other embodiments, port decommissioning maybe performed on both E_ports and F_ports. For clarity, the discussionbelow is typically written in terms of decommissioning E_ports, butsimilar techniques may be used for decommissioning F_ports.

In one embodiment, port decommissioning may be deployed using anout-of-band management methodology, such as one that adheres to theStorage Networking Industry Association (SNIA) Storage ManagementInitiative-Specification (SMI-S) standard, which is incorporated byreference herein in its entirety for all purposes. In such anembodiment, the application may be any type of software package thatuses the SMI-S Common Interface Model (CIM) structure and communicateswith the other components using SMI-S directives. The components areidentified to the application by IP addresses. Where decommissioning ofboth E_ports and F_ports are supported, two lists may be maintained—onefor switches in the fabric (servers) and one for devices that want toevaluate the decommission requests (clients). In embodiments where onlyE_port decommissioning is supported, a single list may be used. FIG. 1demonstrates this deployment context.

FIG. 1 is a block diagram illustrating a deployment of a portdecommissioning technique in a system 100 according to one embodiment.In this embodiment, an application 110 executes on an administrativeworkstation 112 that is connected via a network such as a local areanetwork (LAN) 130 to hosts 140 and devices 150 that are linked by switchfabric 120 of SAN 160. Although illustrated in FIG. 1 as a singlechassis, the switch fabric 120 may comprise a plurality of networkswitches in one or more chassis, connected as a fabric 120 using anydesired connection functionality. In the system 100, hosts 140 anddevices 150 are connected into a SAN using the switch fabric 120. (Thenumbers of hosts 140 and devices 150 are illustrative and by way ofexample only, and any desired number of hosts 140 and devices 150 may beconnected in the system 100.

The application 110 performs the analysis and execution of thedecommission operation. This includes querying the switches in thefabric 120 for the list of ports then sending block requests for theselected ports. It also includes querying the devices 150 or hosts 140that want to evaluate the decommission request to determine if thesubject F_port or F_ports can be decommissioned.

The application 110 may be provided on any desired non-volatilecomputer-readable program storage media including, but not limited to,all forms of optical and magnetic, including solid-state, storageelements, including removable media. The application workstation 112 maybe any desired type of computational device, including a virtualizedcomputer executing on any real hardware platform desired.

In one embodiment, port decommissioning may use a command line interfaceimplemented on the switch 120 to allow an administrator to decommissiona port, using a port identifier as a parameter to the decommissioncommand. The command may block waiting for completion of thedecommission request. The command may result in a message indicatingsuccessful or unsuccessful completion of the decommissioning of theport. Although some embodiments may only allow decommissioning a singleport with a single command, other embodiments may allow decommissioningranges of ports and multiple non-contiguous ports with a single commandif desired.

In various embodiments, one or more conditions may prevent successfuldecommissioning of a port, including (1) the local port is not in aready state, (2) the local port is not marked as a lossless port, (3)there is no redundant path to reach all domains, (4) the remote switchport is not in a ready state, (5) the remote switch port is not markedas a lossless port, (6) the remote switch does not support portdecommissioning, (7) the port is the last traffic isolation path in afailover disabled traffic isolation zone, and (8) a decommission requestis already in progress for this port. Some of these conditions may onlyapply to E_ports, while others may only apply to F_ports, and others mayapply to any port.

In one embodiment, an E_port or an F_port of a network switch may bedecommissioned if disabling that port would not make unreachable anydestination that was reachable by any node of the fabric 120 prior tothe decommissioning. In other embodiments, an E_port may bedecommissioned if disabling that port will not cause any domainreachable from that E_port to become unreachable, even if the routingcosts to those domains may change. In yet other embodiments, an E_portmay only be decommissioned if the redundant route is of equal or lesserrouting cost.

In one embodiment, an E_port that is not part of a trunk (linkaggregation) group may be decommissioned if it is in an operationalstate with at least one other redundant E_port of equal cost availableto reach all domains that the E_port to be decommissioned can reach.

If the E_port is a member of a trunk group, the E_port may bedecommissioned if there is at least one other member in the trunk group.When a trunk port is decommissioned, frames will use other ports in thetrunk group to transmit. In one embodiment, if the E_port is the lastoperational E_port in a trunk group, and there are other availableredundant E_ports, then the E_port and trunk group may be decommissionedand taken offline. If there are no available redundant E_ports, then thedecommissioning may be rejected if the E_port is the last operationalport in the trunk group.

In one embodiment, to be considered a redundant path for the purposes ofdecommissioning an E_port, the path must be an E_port connection betweenthe same pair of switches as the E_port being decommissioned. Similarly,in order to the considered a redundant path for the purposes ofdecommissioning an F_port, the path must be a connection between anotherfabric port of the network switch and another node port of the device.

Upon successful completion of the command all traffic flows on theE_port will have been moved to other redundant paths and the port willbe placed in a blocked state. To bring the port back online theadministrator may re-enable the port.

In another embodiment, an E_port may be decommissioned through aninterface provided by a fabric management system that may provide agraphical user interface (GUI) for controlling and managing one or moreswitch fabrics of an enterprise.

In one embodiment, if an administrator attempts to decommission bothends of an ISL simultaneously, the system may accept one of thedecommissioning requests and reject the other request. In a furtherembodiment, the selection of which decommissioning request to process byaccepting the request to decommission the port in the higher domain,rejecting the request in the lower domain. This allows the lower domainto wait until the higher domain decommissions the port.

Port decommissioning is a function that allows an administrator toremove a port or set of ports from service in a manner that insures thatthe operation does not affect active environments. This is achieved bycoordinating the decommission request with those switches or devicesusing the port or ports as a resource. Requests to decommission F_Portsare coordinated with the applicable devices and requests to decommissionE_ports are coordinated with the applicable switches. As such, thedevices or switches have the opportunity to confirm or deny the requestbefore the decommission action is executed.

FIG. 2 is a graph illustrating a state diagram for the high-levelactions that are performed to achieve port decommissioning according toone embodiment. In this embodiment, a user 210 selects a port todecommission from a display of a fabric topology; other embodiments mayrequire the user 210 to have previously determined a port todecommission. FIG. 2 is set forth as if the port to be decommissioned isan F_port, connected to a device 230, but a similar procedure may beperformed if the port to be decommissioned is an E_port, connecting aswitch to another switch in the fabric 120.

To begin, a user 210 selects the port decommission function to theapplication 220 in action 212, either by way of a command line interfaceor through a management system user interface. The application 220 thenreads the fabric topology in action 222, requesting that informationfrom the fabric 240. The fabric 240 presents fabric topology informationto the application 220 in action 242, using any desired representationof the topology, allowing the application to display the fabric topologyfor the user in action 224, using any desired technique for displayingthe topology, including a graphical display of the fabric topology.

The user 210 may then select a port to decommission, indicating thatselection to the application 220 in action 214. The application 220 maythen issue a decommission request to the device 230, which performs thedecommissioning. The device 230 confirms to the application 220 inaction 232 that the port can be decommissioned, or that decommissioningis not possible at that time.

If the decommissioning cannot be performed, the application 220 mayprovide the user 210 with information indicating the reason provided bythe device 230 that the port or ports cannot be decommissioned in action228. When a request to decommission an ISL is presented, the twoswitches in the fabric 240 involved in the transaction assume the roleof devices 230 within the context of the port decommissioning operation.In that case, the switches move the flows assigned to the ISL beforeconfirming the decommission request. If the flows cannot be moved, theswitch declines the request with the appropriate reason code.

If the decommissioning is confirmed by the device 230, the application220 issues a block request to the fabric 240 in action 226. In anembodiment that allows the user to request decommissioning of multipleports, only the ports that have been confirmed by the device 230 areblocked. The fabric 240 then provides in action 244 the status of theblock request for the port or ports being decommissioned. Finally, theapplication 220 displays to the user 210 the status of thedecommissioning in action 228, completing the decommissioning request.

As stated previously, in one embodiment port decommissioning isperformed by functionality of a management system, such as the DataCenter Fabric Manager (DCFM®) provided by Brocade CommunicationsSystems, Inc. (DCFM is a registered trademark of Brocade CommunicationsSystems, Inc.) In such a management system, port decommissioningfunctionality may be implemented as a user interface element thatpresents a user with a sequence of dialog boxes that lead the userthrough a series of steps to accomplish the desired result, commonlyreferred to as a wizard.

In such an embodiment, once the user selects the decommission operation,the wizard guides the user through the necessary steps of the operation.FIG. 3 is a flowchart illustrating a technique for decommissioning aport implemented by a wizard according to one embodiment. In block 310,the wizard allows the user to select the switch or switches containingthe port(s) to be decommissioned. The wizard presents the portsavailable for decommissioning on the selected switch(es) and in block320, the wizard allows the user to select the port(s) to bedecommissioned. The wizard may then obtain information about eachselected port and construct a list of Fibre Channel identifiersassociated with the selected port(s) in block 330. In block 340, thatlist may be distributed to each of the devices participating in theselected port(s).

If the devices currently support decommissioning of the selectedport(s), then in blocks 350-370, the wizard may display the confirmationstatus for each selected port. If a selected port is an F_port, asidentified in block 350, then that status may be that the request isconfirmed, denied because the port is in use, or denied because the portis the last port of the device. If the selected port is an E_port, thenthe status may be that the request is confirmed, denied because the portis in use, denied because the switch cannot isolate the port, or deniedbecause the E_port is part of the last path to the remote switchconnected by that ISL.

If the selected port is confirmed for decommissioning, then in block 380the wizard may issue the request to the switch containing each port toblock that port. Finally, in block 390, the wizard may display thestatus of the selected port, typically indicating the port as blocked.

In an embodiment employing a wizard in a management system, the wizardmay provide options for handling the variations and exceptions thatarise when executing a port decommissioning operation. The type ofvariations and exceptions may vary depending on whether the request isdeclined by a device for an F_port or a switch for an E_port.

For example, a participating device 150 may decline the request todecommission an F_port if the port is still in use or is the last accesspoint into the fabric 120 for a multi-ported device. If a decommissionrequest is declined by a device 150, the wizard may offer the user theoption of terminating or forcing the operation in one embodiment. If theoperation is terminated and a portion of the function was successful(e.g., some ports of a selected group of ports were decommissioned), thewizard may undo the successful functions by unblocking the ports thatwere blocked. If the user chooses to force the operation, the wizard maysimply block those ports that the device 150 declined to decommissiongracefully.

In another example, a participating switch in one embodiment may declinethe request to decommission an E_port if the flows using the port cannotbe moved to another ISL or if the ISL is the last path into the fabric120 and thus decommissioning would result in fabric isolation. If adecommission request is declined by a switch, the wizard may offer theuser the option of terminating or forcing the operation. If theoperation is terminated and a portion of the function was successful, aswith declined requests to decommission an F_port, the wizard may undothe successful functions, unblocking ports that were blocked. If theuser chooses to force the operation, the wizard may simply block thoseports that the switch declined to decommission gracefully.

In one embodiment, decommissioning of an E_port may be implemented inpart by the Fabric Shortest Path First (FSPF) module of an operatingsoftware of the switches that comprise the fabric 120. In order to bringdown an online redundant E_port without traffic disruption, both sidesof the ISL must stop using the port to transmit data and the transmitbuffers for the port must be flushed before the port is brought offline.To accomplish this, the FSPF module may implement a protocol to sendUNREACHABLE events to the routing engine (RTE) of the switch for theports at each end of the ISL at staggered time intervals before theports are disabled. The UNREACHABLE event tells the RTE that theneighboring switch has become unreachable from the E_port that is beingdecommissioned. This protocol is described in more detail below in thediscussion of FIG. 4.

The UNREACHABLE events for the E_ports on both ends of the ISL must notbe sent at the same time because UNREACHABLE events sent to the RTE maycause the switch to be locked, so new inbound frames cannot be queued tothe switch. This also typically causes the switch to stop issuing theR_RDY link primitive. If both switches on each end of the ISL are lockedat the same time, the switch will not drain any existing frames on theport and this may result in out of order frames when the route is moved.In order to avoid locking the switches at the same time, portdecommissioning may use a chassis lock protocol.

In one embodiment, the FSPF module will ensure that the port beingdecommissioned is not the last path in an active failover disabledtraffic isolation (TI) zone, as this would cause traffic disruptions.Traffic Isolation dedicates paths in the fabric 120 to specific dataflows, enabling predictability and avoiding network congestion. Trafficisolation is implemented using a special zone, called a TrafficIsolation zone (TI zone). A TI zone indicates the set of N_ports andE_ports to be used for a specific traffic flow. When a TI zone isactivated, the fabric attempts to isolate all inter-switch trafficentering from a member of the zone to only those E_ports that have beenincluded in the zone. The fabric also attempts to exclude traffic not inthe TI zone from using E_ports within that TI zone. If failover isdisabled and a TI-zoned route fails, traffic for that TI zone halts. Ifthe user attempts to decommission a port that is the last path in afailover disabled TI zone, the FSPF module may return an appropriatestatus code and inform the user that the port cannot be decommissioneddue to the current state of the TI zone.

In one embodiment, the FSPF module may implement a public libraryfunction for management entities to call to request a port bedecommissioned. The library function may make an interprocesscommunication call into an FSPF daemon. The call may set a timeout for apredetermined timeout expiration and not return until either the porthas been decommissioned, the decommission request has failed for somereason, or a timeout has occurred. If the request fails, a status valuemay be returned to indicate the failure reason.

A chassis lock protocol according to one embodiment consists of a singlechassis lock across switch instances on a chassis. In order to sendreachable/unreachable messages to the RTE for port decommissioning, inone embodiment a port must hold the chassis lock for his own chassis andthe neighbor chassis. When a port attempts to acquire the locks neededfor programming the RTE, it first attempts to obtain its own chassislock and then sends a lock request frame to the remote FSPF. If the porthub is unable to obtain its own chassis lock or the lock request frameto the remote FSPF fails, the port may place itself in a timer statewith a random back-off time and retry when the timer expires. Thisdesign ensures that only one port on any switch can be locked at onetime and prevents any deadlocks from occurring.

In one embodiment, a new Fibre Channel Switch Internal Link Service(SW_ILS) command is defined to send a decommission request to a neighborswitch. The command, Request Port Decommission, may be sent as anasynchronous SW_ILS request. The response may be a switch accept(SW_ACC), with a code indicating that the request has been accepted ordenied; alternately, the response may be a SW_ACC if the request hasbeen accepted and a switch reject (SW_RJT) if the request has beendenied. The payload for the Request Port Decommission command maycontain the originator and recipient domain ID and the local and remoteport numbers. The rejection code may indicate that the remote port isnot in a ready state, is not enabled for lossless communications, noother redundant E_port path is available, or that the port is the lastpath in a failover disabled TI zone.

FIGS. 4 and 5 are sequence diagrams that describe the decommissioningbehavior of a port decommissioning protocol according to one embodiment.When a request to decommission a port is received by the FSPF module,the FSPF module checks that the port is an E_port in the DONE state,that another available redundant E_port exists, and that losslesscommunication is enabled. If these conditions are met, the FSPF modulesends an asynchronous Request Port Decommission request to the neighborFSPF module. The FSPF module also saves the IPC callback information sothat the caller can be notified when the decommission request iscomplete or fails for some reason. FIG. 4 is a sequence diagram thatdescribes a protocol for decommissioning a non-trunk E_port, while FIG.5 is a sequence diagram that describes a protocol for decommissioning atrunk E_port. In one embodiment, the last port in a trunk group may beE-port decommissioned as if it is a non-trunk E_port.

To begin the decommissioning of a non-trunk E_port, management system410 sends a port decommission request 411 to FSPF module 420. FSPFmodule 420 sends a decommission request to FSPF module 440 of theneighboring switch. FSPF module 440 sends a decommission requestaccepted response 441, accepting the request. FSPF 420 then sends arequest to obtain the chassis lock 422 to the FSPF 440, which isresponded to by a lock request accepted 442. The FSPF 420 then sends anunreachable event 423 to the local switch driver 430, and then sends alock release request 424 to the remote FSPF 440. The remote FSPF 440accepts the lock release request 424, responding with a lock releaseaccepted 443. The remote FSPF 440 then sends a request 444 for thechassis lock of the local chassis to the local FSPF 420. The local FSPF420 response with a lock request accepted 425. Upon receiving the lockrequest accepted 425, the remote FSPF 440 sends an unreachable event 445to the remote switch driver 450. The remote FSPF 440 may then send arequest to release the lock 446 to the local FSPF 420, which respondswith a lock release accepted 426. The FSPF 420 may then send a request427 to block the local port to the switch driver 430, before respondingwith a decommissioning success indication 428 to the management entity410.

To begin the decommissioning of a trunk E_port, management system 410sends a port to decommission request 511 to FSPF module 420. FSPF module420 sends a decommission request 521 to FSPF module 440 of theneighboring switch. FSPF module 440 then sends a request 541 to theremote switch driver 452 to credit returns for the member of the trunkgroup being disabled. The remote FSPF 440 response to the decommissionrequest 521 with a decommission accepted message 542 that is returned tothe local FSPF 420. The local FSPF 420 may then send a request 522 tothe local switch driver 430 to stop credit returns for the member of thetrunk group being disabled, followed by an instruction 523 to disablethe decommissioned E_port. Both sides must disable returning credits toforce the other side to stop sending traffic across the trunk member tobe decommissioned. The trunk member cannot be disassociated from thetrunk group while one side is still sending traffic otherwise frameswill be dropped. Once both sides have successfully stopped returningcredits on that member, the initiator disables the port. Both sides ofthe link then may update the trunk members and the trunk bandwidth. Uponcompletion of disabling the E_port by the local switch driver 430, thelocal switch driver 430 may send an off-line/master change state changenotice (SCN) 531. The remote switch driver 450 may also send anoff-line/master change SCN 551 to the remote FSPF module 440.

Upon receipt of the off-line/master change SCN 531, the local FSPF 420may send a request for a chassis lock 524 to the remote FSPF 440, whichresponds with a lock request accepted 543. The local FSPF 420 may thensend a message 525 to the local switch driver 430 to update thebandwidth for the trunk group containing the E_port, then send a lockrelease message 526 to the remote FSPF 440.

Upon receipt of the lock release 526, the remote FSPF 440 responds witha lock release accepted message 544, and then sends a request for thechassis lock 545 to the local FSPF 420. The local FSPF 420 indicatesacceptance of the chassis lock request 527. The remote FSPF 440 may thensend a message 546 to the remote switch driver 450 to update thebandwidth for the trunk group, and then send a lock release message 547to the local FSPF 420. The local FSPF 420 may then release the lock,sending a lock release accepted 528 back to the FSPF 440. Finally, thelocal FSPF 420 may indicate the E_port has been successfullydecommissioned 529 to the management system 410.

In one embodiment, the FSPF module uses a state machine 600 for portdecommissioning. Where a state machine is used for losslessinitialization of the port, the decommissioning state machine may be anextension of the lossless initialization state machine.

As indicated by the differences in the sequence graphs of FIGS. 4 and 5,the behavior for a port that is part of a trunk group going offline isdifferent from that of a port that is not part of a trunk group, becauseno UNREACHABLE needs to be sent to the RTE for the trunked port. Theport is removed from the trunk, via the call to the switch driver IOCtl,credit returns are stopped, and then the port can be disabled. Afterthis, both ends of the link will update the bandwidth of the trunkgroup.

In one embodiment, if the user issues the command at both ends of an ISLat the same time, only one request may be accepted. An arbitrationmechanism may be provided that uses domain IDs for deciding whichrequest to accept. Thus, the higher domain may reject the lower domain'sdecommission request if it has already issued a decommission request.The lower domain may accept the decommission request. This arbitrationmechanism is illustrative and by way of example only, and othertechniques may be used for deciding which request to perform.

In one embodiment, coordinating the decommissioning of an F_port of anetwork switch with a device 150 may be implemented by sending a requestfrom the network switch to the device 150 to decommission the F_port.The device 150 may determine whether to accept the request and respondwith acceptance or rejection as desired. Upon receiving the acceptance,the network switch may block the F_port.

An Example Hardware Implementation

In one embodiment illustrated in FIG. 6, the hardware functionality forthe port decommissioning features described above is implemented as amultiport Fibre Channel switch application specific integrated circuit(ASIC) 610 that is combinable with a processor subsystem 620 to providea complete multiport Fibre Channel network switch 600. Multiple ASICs610 can be arranged in various topologies to provide higher port count,modular switch chassis. The ASIC 610 and processor subsystem 620 areillustrative and by way of example only, and other hardwareimplementations can be used as desired.

The ASIC 610 comprises four major subsystems at the top-level as shownin FIG. 6: A Fibre Channel Protocol Group Subsystem 630, a Frame StorageSubsystem 640, a Control Subsystem 650, and a Processor System Interface660. Some features of the ASIC 610 that are not relevant to the currentdiscussion have been omitted for clarity of the drawing.

The Fibre Channel Protocol Group (FPG) Subsystem 630 comprises port andserialization/deserialization (SERDES) logic blocks to provide E, F, andFL ports.

The Frame Data Storage (FDS) Subsystem 640 contains the centralizedframe buffer memory and associated data path and control logic for theASIC 610.

The Control Subsystem 650 comprises the switch control path functionalblocks, including frame routing.

The Processor System Interface 660 provides a programming interface tothe ASIC 610. As illustrated in FIG. 6, the ASIC 610 is connected to theProcessor Subsystem 620 via a PCIe link but other architectures forconnecting the ASIC 610 to the Processor Subsystem 620 can be used.

Some functionality described above can be implemented as softwaremodules in an operating system or application running on a processor 622of the processor subsystem 620 and stored in a memory 624 or otherstorage medium of the processor subsystem 620. This software may beprovided during manufacture of the switch chassis 600, or provided onany desired computer-readable medium, such as an optical disc, andloaded into the switch chassis 600 at any desired time thereafter. Thistypically includes functionality such as the software that allows thecreation and management of logical ports that are defined for the ASIC610 and logical ISLs (LISLs) to connect logical ports, as well as userinterface functions, such as a command line interface for management ofthe switch chassis 600.

In one embodiment, the control subsystem 650 is configured by operatingsystem software of the network switch 600 executing in the processor 622of the processor subsystem 620.

Serial data is recovered by the SERDES of an FPG block 630 and sent tothe FPG subsystem 630, which is responsible for performing decoding,cyclic redundancy check (CRC) checking, minimum and maximum lengthchecks, disparity checks, etc. The FPG subsystem 630 sends the frame tothe Frame Storage System (FDS) subsystem 640. The location where theframe is stored is passed to the control subsystem, and is used as thehandle of the frame through the ASIC 610. The Control subsystem 650performs routing, classification, and queuing functions on the frame.Frames are queued on transmit ports based on their routing, filteringand Quality of Service (QoS). Transmit queues de-queue frames fortransmit when credits are available to transmit frames. When a frame isready for transmission, the Control subsystem 650 de-queues the framefrom a transmit queue for sending out through the FPG 630.

The control subsystem 650 provides processing with a variety ofapplications through a programmable interface to software, including (a)Layer2 switching, (b) Layer3 routing (FCR) with complex topology, (c)LUN remapping, (d) LUN zoning, (e) Hard zoning, (f) VSAN routing, (g)Selective egress port for QoS, and (g) End-to-end statistics.

The control subsystem 650 provides hardware capable of encapsulating androuting frames across inter-switch links that are connected to the ports630 of the ASIC 610, including the transport of logical ISL frames thatare to be sent across an extended ISL (XISL). The control subsystem 650performs frame header processing and Layer 3 routing table lookupfunctions using routing tables where routing is required, encapsulatingthe frames based on the routing tables, and routing encapsulated frames.The control subsystem 650 can also bypass routing functions where normalLayer2 switching is sufficient.

Thus, the ASIC 610 can use the control subsystem 650 to perform theencapsulation, routing, and decapsulation, by adding or removing headersto allow frames for a LISL to traverse an XISL between network switchesas described above at hardware speeds.

CONCLUSION

By automatically coordinating the decommissioning of ports in a switch,an administrator may improve the reliability and performance of theswitch fabric, ensuring that ports are gracefully taken out of servicewithout unnecessary interruption of service and triggering of automaticrecovery functionality that may occur during manual decommissioning ofports. Embodiments may provide for decommissioning of F_ports only,E_ports only, or both F_ports and E_ports. Where trunking is providedfor in the fabric, decommissioning of a port in a trunk group may beperformed if there are other operational links in the trunk group.Decommissioning of a non-trunked port (or the last operational port in atrunk group) may be performed if there is a redundant path available, toavoid data loss or out of order transmission in lossless environments.

Although described above generally in terms of FC fabrics and using FCterminology, the problems and techniques for graceful decommissioning ofports are not limited to FC fabrics and protocols, but may beimplemented using similar techniques to those described above in FibreChannel over Ethernet (FCoE), Ethernet, and other types of networks,including lossless networks using IEEE Data Center Bridging.

It is to be understood that the above description is intended to beillustrative, and not restrictive. For example, the above-describedembodiments may be used in combination with each other. Many otherembodiments will be apparent to those of skill in the art upon reviewingthe above description. The scope of the invention therefore should bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled. In the appendedclaims, the terms “including” and “in which” are used as theplain-English equivalents of the respective terms “comprising” and“wherein.”

1. A method, comprising: receiving a request to decommission a firstport of a first network switch; coordinating decommissioning of thefirst port with another network element connected to the first networkswitch via the first port; and rejecting the request unless everydestination reachable from the another network element would remainreachable after decommissioning the first port.
 2. The method of claim1, further comprising: rejecting the request unless the another networkelement is reachable via a second port of the network switch.
 3. Themethod of claim 1, wherein the first port comprises a switch port andthe another network element comprises a second network switch.
 4. Themethod of claim 1, wherein the first port is configured as a member of alink aggregation group.
 5. The method of claim 1, wherein the act ofcoordinating decommissioning a first port of a first network switchcomprises: sending a decommission request to a device connected to thenetwork switch via the first port; receiving acceptance of thedecommission request from the device; and blocking the first port of thenetwork switch.
 6. The method of claim 1, wherein the act ofcoordinating decommissioning a first port of a first network switchcomprises: sending a decommission request to a second network switchconnected to the first network switch via the first port; receivingacceptance of the decommission request from the first network switch;indicating that the second network switch is unreachable via the firstport; indicating that the first network switch is unreachable via thesecond port of the second network switch; and blocking the first port ofthe first network switch.
 7. The method of claim 6, wherein the act ofcoordinating decommissioning a first port of a first network switchfurther comprises: obtaining a lock associated with the second networkswitch before performing the act of indicating that the second networkswitch is unreachable via the first port; and obtaining a lockassociated with the first network switch before performing the act ofindicating that the first network switch is unreachable via the secondport of the second network switch.
 8. The method of claim 1, wherein theact of coordinating decommissioning a first port of a first networkswitch comprises: indicating that the first port is unusable for a linkaggregation group of which the first port is a member; and disabling thefirst port.
 9. The method of claim 8, wherein the another networkelement is a second network switch, and wherein the act of coordinatingdecommissioning a first port of a first network switch furthercomprises: obtaining by the first network switch a lock associated withthe second network switch; updating by the first network switch a firstbandwidth value associated with the link aggregation group; obtaining bythe second network switch a lock associated with the first networkswitch; and updating by the second network switch a second bandwidthvalue associated with the link aggregation group.
 10. The method ofclaim 1, wherein the act of receiving a request to decommission a firstport of a first network switch comprises: receiving a request todecommission a plurality of ports of the first network switch.
 11. Anon-transitory computer readable medium on which are stored instructionswhich when executed by a computer perform actions comprising: requestingdecommissioning of a first port of a network switch, wherein thedecommissioning is rejected by the network switch unless any destinationreachable via the first port prior to decommissioning is also reachablevia a second port of the network switch.
 12. A non-transitory computerreadable medium on which are stored instructions which when executed bya network switch perform actions comprising: receiving a request todecommission a first port of a first network switch; coordinatingdecommissioning of the port with another network element connected tothe first network switch via the first port; and rejecting the requestunless every destination reachable by the another network element wouldremain reachable after the first port is decommissioned.
 13. A networkswitch, comprising: a processor; an application specific integratedcircuit coupled to the processor, comprising: a first port, configuredfor transmission of data to another network element linked to thenetwork switch via the first port; software executed by the processor,comprising: logic to receive a request to decommission the first port;logic to determine whether the first port can be decommissioned; andlogic to coordinate decommissioning of the first port with the anothernetwork element.
 14. The network switch of claim 13, wherein the firstport comprises a switch port and the another network element comprisesanother network switch.
 15. The network switch of claim 13, wherein thefirst port comprises a fabric port and the another network elementcomprises a device.
 16. The network switch of claim 13, wherein thefirst port is configured as a member of a link aggregation group. 17.The network switch of claim 13, wherein the logic to determine whetherthe first port can be decommissioned comprises: logic to reject therequest to decommission the first port unless the another networkelement is reachable at the second port of the network switch.
 18. Thenetwork switch of claim 13, wherein the application specific integratedcircuit further comprises a second port, and wherein the logic todetermine whether the first port can be decommissioned comprises: logicto accept the request to decommission the first port if the first portand the second port are configured as members of a link aggregationgroup and the second port is operational.
 19. The network switch ofclaim 13, wherein the logic to coordinate decommissioning the first portwith the another network element comprises: logic to send a decommissionrequest to a device reachable via the first port; logic to receiveacceptance of the decommission request from the device; and logic toblock the first port of the network switch.
 20. The network switch ofclaim 19, wherein the logic to coordinate decommissioning the first portwith another network element further comprises: logic to indicate thatthe another network element is unreachable from the first port.
 21. Thenetwork switch of claim 13, wherein the logic to coordinatedecommissioning of the first port with the another network elementcomprises: logic to send a decommission request to a second networkswitch; logic to stop credit returns associated with a member of a linkaggregation group corresponding to the first port; logic to disable thefirst port; and logic to update a bandwidth of the link aggregationgroup.
 22. A network switch, comprising: a processor; an applicationspecific integrated circuit coupled to the processor, comprising: afirst port; and software executed by the processor, comprising: logic toaccept a decommissioning request from another network switch; logic toaccept a chassis lock request from the another network switch; logic torequest a chassis lock request of the another network switch; and logicto indicate the another network switch is unreachable via the firstport.
 23. A network switch, comprising: a processor; an applicationspecific integrated circuit coupled to the processor, comprising: aplurality of ports grouped as a link aggregation group; and softwareexecuted by the processor, comprising: logic to accept a decommissionrequest from another network switch; logic to accept a chassis lockrequest from the another network switch; logic to stop credit returnsassociated with a member of the link aggregation group responsive toacceptance of the decommission request; logic to request a chassis lockof the another network switch; and logic to update a bandwidth of thelink aggregation group by the network switch and the another networkswitch.
 24. A network, comprising: a first network switch, comprising: afirst switch port; a second network switch, comprising: a second switchport; an inter-switch link, associated with the first switch port andthe second switch port; and a management software, comprising: logic toreceive a request to decommission the first switch port; logic torequest the first network switch to decommission the first switch port;logic to coordinate decommissioning of the first switch port with thesecond switch; and logic to reject the request if any destinationreachable from any node of the network would become unreachable afterdecommissioning the first switch port.
 25. The network of claim 24,further comprising: a management workstation, wherein the logic toreceive a request to decommission the first switch port comprises: logicto send the request to decommission the first switch port from themanagement workstation to the first network switch; logic to receive therequest to decommission the first switch port from the managementworkstation by the first network switch; and logic to indicate a resultof the request to decommission the first switch port.
 26. The network ofclaim 24, wherein the management software further comprises: logic toreject the request unless an operational redundant link exists between athird switch port of the first network switch and a fourth switch portof the second network switch.
 27. The network of claim 26, wherein thefirst switch port and the third switch port are members of a linkaggregation group.
 28. The network of claim 24, wherein the logic tocoordinate decommissioning of the first switch port with the secondnetwork switch comprises: logic to send a decommission request to thesecond network switch; logic to receive acceptance of the decommissionrequest from the second network switch; and logic to block the firstswitch port of the first network switch.
 29. The network of claim 28,wherein the logic to coordinate decommissioning the first switch portwith the second network switch further comprises: logic to indicate thatthe second network switch is unreachable via the first switch port; andlogic to indicate that the first network switch is unreachable via thesecond switch port of the second network switch.
 30. The network ofclaim 29, wherein the logic to coordinate decommissioning the firstswitch port with the second network switch further comprises: logic toobtain a lock associated with the second network switch beforeindicating that the second network switch is unreachable via the firstport; and logic to obtain a lock associated with the first switch beforeindicating that the first network switch is unreachable via the secondport of the second network switch.
 31. The network of claim 24, whereinsome of the management software executes on the first network switch andsome of the management software executes on the second network switch.32. A network, comprising: a network switch, comprising: a fabric port;a device, comprising a node port; a link associated with the fabric portand the node port; and a management software, comprising: logic toreceive a request to decommission the fabric port; logic to request thenetwork switch to decommission the first port; logic to coordinatedecommissioning of the fabric port with the device; and logic to rejectthe request unless every destination reachable from the device wouldremain reachable from the device after decommissioning the fabric port.33. The network of claim 32, wherein some of the management softwareexecutes on the network switch.
 34. The network of claim 32, wherein themanagement software further comprises: logic to reject the requestunless an operational redundant link exists between a second fabric portof the network switch and a second node port of the device.
 35. Thenetwork of claim 32, wherein the logic to coordinate decommissioning ofthe fabric port with the device comprises: logic to send a decommissionrequest to the device; logic to receive acceptance of the decommissionrequest from the device; and logic to block the fabric port.